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ABSTRACT 

In 1986, the Florida State Legislature passed a law 
requiring teachers to demonstrate knowledge of their subject areas 
before being granted teacher certification. The state also provided 
funding for developing subject area examinations. The generic 
procedures for developing these examinations are outlined. During the 
first phase of the project, an Examination Development Committee 
(EDC) determined which competencies and skills teachers should have 
in their subject areas. The committee also recommended percentages of 
the examinacion to be allocated to each competency and skill. Using 
the specified competencies, test item specifications were formulated. 
The second phase of the project entailed the development, testing, 
and refinement of 480 test items. Once developed by the EDC, items 
wera evaluc.Led by a Validation Committee and external sources. 
Revised items were computer arranged into five examination forms and 
administered to first-year teachers and senior-level students 
enrolled in teacher preparation programs in the state of Florida. 
Results were analyzed statistically, and final revisions were made. 
Two tables present examination plans. (Author/SLD) 
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ABSTRACT 

In 1986, the Florida Legislature passed a law requiring 
teachers to demonstrate knowledge of their subject area before 
being granted teacher certification. Funding was provided to 
develop subject area examinations in each teaching specialization. 
In this paper, the generic procedures for developing these teacher 
certification subject area examinations are outlined. 

During the first phase of the project an Examination 
Development Co}Timittee determined which competencies and skills 
teachers should have in their subject area. This committee also 
recommended percentages of the exam to be allocated to each 
competency and skill in a blueprint for exam construction « Using 
the specified competencies, test item specifications were 
formulated. 

The second phase of the project entailed the development, 
testing and refinement of 480 test items. Once developed by the 
Examination Development Committee, items were evaluated by a 
Validation Committee and external sources. Revised items were 
computer arranged into five examination forms and administered to 
first-year teachers and senior-level students enrolled in teacher 
preparation programs in the state of Florida, Re^^ults wore 
statistically analyzed and final revisions made. 
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INTRODUCTION 

Since the publication of A Nation At Risk and the Holmes 
Report describing the inadequacy of public education in the United 
States, various state legislative bodies have proposed numerous 
changes in their requirements for teacher certification. This 
focus is reflective of the perception that quality teaching resultc 
in quality learning. Thu3, adequately prepared teachers are 
essential if students are to learn. 

Many of the legislative enactments have focused on influencing 
the curricula of teacher preparation programs or evaluating the 
product after completion of these programs. According to the April 
20, 1988 Chronicle of Higher Education^ since 1986 thirteen states 
have passed or enacted legislation requiring teachers to pass a 
state-developed subject area knowledge examination prior to being 
granted initial certification. Several other states use the 
National Teacher Examination or a combination of the two for this 
purpose. 

In 1986 the Florida Legislature passed a law (Florida Statue 
#231.17) requiring that all teachers demonstrate knowledge in their 
subject area via examination prior to initial teacher 
certification. This certification criterion was one of several 
which also included passing a basic pedagogical exam and 
successfully completing a one-year Beginning Teacher program. 
Through these criteria, Florida was seeking to meet a goal set 
forth in 1986 publication Tomorrow's Teachers; A Report of the 
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Holmes Group which dealt with the creation of "standards of entry 
to thG teaching profession that are professionally relevant and 
intellectually defensible." 

The purpose of this paper is to describe the procedural 
development of the subject area certification exams for teachers in 
the state of Florida . It is believed that this was the largest 
test development project to be conducted at one time by any state 
department of education. Forty-six subject area tests were 
developed simultaneously under contract to various state 
educational institutions. All subject area projects were conducted 
following the same general guidelines. 

The test development process had two phases: Phase I consisted 
of the development and validation of 1) identified teaching 
competencies and skills for the subject area; 2) a blueprint which 
described the general weighting for each skill and the behavioral 
level for the actual test items; and finally, 3) the development of 
test items specifications from which the actual items would be 
written. Phase II involved the development, pilot testing and 
refinement of an initial 480 test items. 

The following procedural description details the development 
of the subject area examinations for health education and physical 
education. 
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PHASE I PROCEDURES 

In November of 1986, the College of Health and Human 
Performance at the University of Florida contracted with the 
Florida Department of Education to develop tests in the subject 
areas of Health Education K-12 and Physical Education Grades K-8 
and 6-12. (Florida is among those states which endorse 

certification on two levels in Physical Education. ) Identical 
procedures were followed by all subject area test development 
groups . 

The initial task was to identify the competencies and skills 
necessary for beginning teachers in each subject area. In 
structuring this task, three general principles were adhered to: 

1) experienced and informed subject area specialists would 
develop the list^ of competencies and skills. 

2) selected subject area specialists would include 
representatives of teacher educators, supervisors and 
practitioners, minority and union groups, professional 
organizations and all geographic regions of the state. 

3) the competencies and skills identified by the development 
team would be job~related, relevant to successful 
training and measurable. 

Two working committees were established to perform the 
necessary tasks --the Examination Development Committee and the 
Validation Committee. In determining membership on these 
committees, consideration was given to the three aforementioned 
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general principles and recommendations from the Florida Department 
of Education, professional organizations and the two teachers 
unions in the state. 

Initially, the Examination Development Committee (EDC), 
composed of 11 members, was convened to discuss techniques for 
interviewing practitioners in the subject area and reviewing the 
literature. Open-ended interviews were to be conducted by the EDC 
to gather information about the competencies and skills which the 
practitioners believed were necessary for teaching in their subject 
area. The Department of Education (DOE) determined the number of 
interviews for each subject area based upon the number of teachers 
currently active in the field. The instrument was designed and 
standardized by the DOE for use in all subject areas. Team members 
were provided interviewing and recording instructions. Although 
the written instrument was designed to elicit general oral 
responses to questions, space was provided to allow for 
documentation of further comments by the interviewees. The 
categories for response were content knowledge, curriculum, 
instructional media, methods and strategies, evaluation, current 
trends, and classroom management. 

The instrument and examples of categorical responses in the 
area of Physical Education are illustrated in Table 1 . The 
completed interview forms were forwarded by the interviewers to the 
project director of the subject area for analysis. 



Insert Table 1 



The review of the literature was conducted by the EDC to 
identity competencies and skills which educational researchers, 
professional organizations or state departments of education had 
previously determined to be essential for teachers in their 
particular subject area. The scope of scholarly publications 
reviewed included current texts, professional journals, state 
designed curriculum frameworks, and state and national professional 
organizations' guidelines for educationally sound programming. 

A subcommittee of the EDC consisting of the project director 
and two other EDC members comprised a Core Team which was primarily 
responsible for initial development and subsequent revisions of all 
documents produced. Using information gathered from the literature 
review, the analysis of the personal interviews and their own 
professional expertise, the Core Team developed the first draft of 
the competencies and skills to be addressed in the examinations. 

The entire EDC was reconvened to review Draft One and to make 
recommendations for revisions. Every effort was made to encourage 
discussion and careful analysis of the identified 
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competencies and skills. Suggested changes were made, resulting in 
Draft Two. 

The next step involved an internal review by the eleven-member 
Validation Committee, comprised primarily of practitioners. With 
the Core Team present to act in an advisory capacity, the 
Validation Committee reviewed each competency and 

skill and made suggestions for acceptance, deletion, or 
improvement. During this review the committee also determined if 
the identified skills could be measured in the multiple-choice uest 
format designated by the DOE for the examinations. 

Another charge of the Validation Committee was to recommend 
the weighting percentage each skill should receive on the completed 
examination form. Type or types of questions (recall, application ^ 
or problem-solving) which could best evaluate each skill were to be 
identified. From these discussions a preliminary blueprint for the 
examination was formulated. (An example of the Health Education 
blueprint for the competency area of Drugs/Alcohol/Tobacco is 
illustrated in Table 2.) 



Insert Table 2 



The blueprint consisted of an outline of the competencies and 
skills, the percent allocation for each of these on the examination 
form and a designation as to the type(s) of questions 



to be used to evaluate each skill. Figures in the "Total" column 
represent the percentage of questions from the competency area to 
be included in the examination form. In the example presented in 
Table 2, 11% of the questions would be on Drugs /Alcohol/Tobacco. 

Thus Draft Three of the competencies and skills and an initial 
blueprint based upon majority consensus was formed. After an 
editorial review by an outside source, uhe document was forwarded 
to the Department of Education for an external review. 

The external review of the developed competencies and ski.lls 
was conducted by the University of South Florida under a separate 
contract. Certified teachers, school-based administrators, county- 
level administrator?, and college-level instructors were asked to 
complete a written survey instrument using a five-point scale, with 
"1" indicating no relation and "5" indicating a very high relation 
to the subject area. For each skill, survey the respondent was 
asked to judge four areas: 

1 ) Job relatedness — the degree to which each skill is 
related to actual practice in this field. 

2) Frequency of use — how often each skill is used in this 
f ieldo 

3) Importance for initial certification — value for a new 
teacher in this field. 

4) Important for career achievement/advancement — value for 
a career teacher in this field. 

Space was provided for survey participants to make additional 
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written comments^ which were later considered by the Core Team in 
making adjustments to the list of competencies and skills • 

All surveys w -^.e submitted for statistical analysis to the 
University of South Florida, the designated data coordination unit 
for all projects. Printouts were provided to the project directors 
for use with the Core Team in evaluating the results. The 
information from the external review validation served as the basis 
to modify the competencies and skills and to finalize the 
blueprint . 

Based upon the statistical analysis, it was deemed appropriate 
to examine any item below a pre-determined acceptable mean value 
for deletion or revision consideration. It was further determined 
that the area "importance for initial certification" best complied 
with the purpose of this project ^ and therefore responses in this 
area were given the highest consideration in determining deletions 
or modification of skills. To illustrate, if a skill received a 
"low" value in terms of its relation to actual practice, but was 
rated "high" in importance for initial certification, the rating 
supported retention of the skill in this draft. 

Using the comments and ratings from the survey, the Core Team 
further refined the blueprint to clearly >-eflect the importance of 
each skill as determined by the broad base of subject area 
professionals in the state who responded. Areas for potential 
changes identified by the Core Team were presented in 
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the form of Draft Four at a meeting of the Validation Committee • 
Each validation team member received a copy of the statistical 
analysis and comments. Each skill and Draft Four in its entirety 
was discussed and group consensus was obtained prior to any 
changes. This meeting resulted in the completion of Draft Five of 
the document. 

Once all changes were made and validated, the competencies, 
skills and blueprints underwent a final editorial review, and plans 
were put into act:ion to develop the test item specifications. All 
memb-*rs of the EDC were sent lists of the various competency topics 
several weeks prior to the first specifications development meeting 
and requested to rank order the topics according to their perceived 
personal knowledge in each* Using the information provided from 
this self-ranking, individuals were assigned to competency areas in 
teams of two. 

All committee members were provided with their writing 
assignments in advance and requested to bring any supportive 
materials to facilitate their task. General information about item 
specifications was also mailed out prior to the meeting. A 
training meeting on the development of test item specifications was 
conducted by the Department of Education personnel prior to 
beginning the task. 

The purposes of devaloping item specifications were: 
J ) To provide an operational definition of each skill to be 
measured. 
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2) To serve as a guide for producing test items o 

3) To ensure consistency among authors and over time. 

The specifications were designed to serve as a guide in developing 
these and future examination items and forms to be used in the 
initial certification assessment process. 

The item specifications included: identification of the skill; 
the "stimulus attributes", or information which poses the question; 
and the "response attributes", or information which presents the 
correct cind incorrect foils (answer choices); and a sample item. 
Sufficient specificity in each item specification was needed to 
assure that the generated items would measure the skill. However, 
specifications also had to be sufficiently generic to enable 
several items to be developed from one specification. 

Upon completion of the draft item specifications by the EDC, 
the Validation Team was convened to review each specification and 
suggest any changes to increase the usability of the item 
specification. "^he specifications were then reviewed by the test 
and measurements consultant and a professional copy editor. 

PHASE II PROCEDURES 

Phase II of the project involved the actual development of 480 
test items (400 plus 20%). These items would compose the item bank 
for test forms of 125-150 questions each, y 




Again, two committees were organized, an Item Writing 
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Committee to write the actual test items and a Validdtion Committee 
to review and critique the test items. The first task was to 
identify committee members who were both knowledgeable in content 
and experienced in constructing psychometrically-sound test items. 
The Item Writing Committee was composed primarily of professionals 
involved in teacher preparation who were recognized content 
specialists. The Validation Committee was composed primarily of 
practitioners currently teaching in the field. 

A training session on the specifics of item writing was 
conducted by a test and measurement consultant. Every effort was 
made to ensure that developed test items would be valid and 
reliable. Instruction included a review of the characteristics of 
good test items ^ how to use the item specifications, and how to 
construct or format items. 

Because of the nature of the project a security plan was 
devised and each committee member was requested to sign a security 
agreement. All item development was conducted at one location with 
item writers working in teams of two. Any scrap materials were 
collected and shredded daily. All items developed each day were 
secured overnight and upon completion of test development by the 
responsible institution. 

For pilot test purposes, items were distributed into five 
examination forms. The intent was to allow no individual to view 
and ansv/er more than 100 of the test items. A DOE-designated 
number of senior-level students currently enrolled in teacher 
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preparation programs in the state of Florida or first-year teachers 
were asked to complete a test form in their area of specialization. 
Examinees were encouraged to react to the suitability and level of 
difficulty of the items by writing comments in the examination 
booklet. At the conclusion of the examination session, five 
students were randomly selected to respond orally to open-ended 
questions regarding the validity of the test. 

The pilot tests were computer-scored and statistically 
analyzed by the Office of Instructional Resources at the University 
of Florida. Raw score analysis included the mean, median, mode, 
standard deviation and standard error of measurement based upon 
total group data. The Kuder-Richardson 20 reliability coefficient 
was used to determine test reliability. An analysis of functional 
distractors and indices of difficulty and discrimination indicated 
the difficulty of the items and the ability of the items to 
discriminate between better and poorer students. A point bi-serial 
correlation revealed the degree to which the items correlated to 
the total test score, indicating the items' consistency. 

The project director reviewed the information obtained from 
the pilot test analysis, written comments, and personal interviews 
for the purpose of making recommendations for test item refinement 
by the Validation Team. "Reviewable items" were identified as 
those not falling within acceptable ranges in the 
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correlation analysis. The Validation Team met to review all test 
items and make further revisions based upon the available 
information and their professional expertise. After these 
suggested changes were made, the items were again reviewed by the 
test and measurement consultant and a professional copy editor. 

Recommended changes by the test and measurement consultant and 
the copy editor were submitted for approval to the project 
director. Subsequently, the resultant items and the computer disks 
on which all items were stored were forwarded to the Department of 
Education. 

The Florida Department of Education began administering the 
subject-area tests in the fall of 1988. Initially, tests 
administered during 1988 and 1989 were primarily used for the 
purpose of establishing normative data. Subject-area task forces 
were formed to review the data and determine passing scores for all 
subsequent administrations of the examinations. 

RECOMMEND AT I ONS 

The project was a monumental task requiring an in-depth look 
at our professions . The following recommendations are made to 
assist in the organization of future test development projects of 
this nature. 

1 . Timelines should be confirmed well in advance of the 
initiation of the phase and should allow sufficient time for 
form processing, typing, travel arrangements, and job demands 
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2. Secretarial assistance is critical to the success of a project 
of this magnitude. Consideration should be given to the 
demands during peak production times and additional clerical 
support should be provided. 

3. Each phase should be completed prior to initiation of the next 
phase. 

4 . Summer months are not recommended for committee meeting dates 
due to vacations and other commitments. 

5. Specification writers and item writers should be the same to 
provide continuity. 

6. Phase I should incorporate development of competencies, skills 
and suggested blueprints. Phase II would then include 
specification writing, blueprint finalization and item 
development. 

7. Understanding the total process and the mechanics of writing 
specifications and items would assist the Phase I committee 
members in developing competencies and skills. 

8. The involvement of a measurement expert to conduct training 
sessions in sound examination preparation is recommended prior 
to specification writing. 
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^TABLE 1 
Interview Instrument 



I. Content Knowledge 

Skill 

Demonstrate knowledge and understanding of 
child development. 

n. Chirriculiun 

Skill 

Demonstrate knowledge of which sports and 
activities are appropriate for the grade level, 

in. Instructional Media, Methods & Strategies 

Skill 

Demonstrate currency in knowledge of resources 
available in Physical Education. 



Application 
Initial Cert. 



Application 
Initial Cert. 



Application 
Initial Cert. 



IV. E^^uation 

Skill 

Utilize supportive data to determine grades and 
student progress. 

V. Current Trends 

Skill 

Demonstrate knowledge of state mandates and 
their effect on the cinriculum. 

VI. Classroom Management 

Skill 

Demonstrate a knowledge of management skills 
to provide good class control. 



Application 
Initial Cert. 



Application 
Initial Cert. 



Application 
Initial Cert. 



Vn. other 

Skill 

Demonstrate a knowledge of budgetary procedures- 



Application 
Initial Cert. 
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TABLE 2 
Health Education Blueprint 



Topic. Competency. Skill REC* APP* PRS' 



By 
Skin 



Vm. Dmgs/Alcohol/Tobacco 

A. Effects on health 2 1 

B. Factors associated with 
substance use and abuse I l 

C. Socio-economic and 

legal consequences. 1 l 

D. Signs and symptoms 

cf abuse. 2 0 

E. Fhrevention and control 1 1 



0 

0 

0 
0 



2 

2 

2 
2 



Total 
Comp 
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